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| [ the language of a translation furnished for the purposes of international search (under Rule 23.1(b)). 

| | the language of publication of the international application (under Rule 48.3(b)). 

I I the language of the translation furnished for the purposes of international preliminary examination (under Rules 55.2 and/ 
or 55.3). 

3. With regard to any nucleotide and/or amino acid sequence disclosed in the intemauonal application, the international 
prelirriinary examination was carried out on the basis of the sequence listing: 

□ contained in the international application in printed form. 

| | filed together with the international application in computer readable form. 

| | furnished subsequently to this Authority in written form. 

| | furnished subsequently to this Authority in computer readable form. 

□ The statement that the subseouently furnished written sequence listing does not go beyond the disclosure in the 
international application as filed has been furnished. 
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V. Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial applicability; 
citations and explanations supporting such statement 



1. statement 

Novelty (N) 



Claims NONE 



Claims 1-25 



YES 
NO 



Inventive Step (IS) 



Claims 
Claims 



NONE 



1-25 



YES 
NO 



Industrial Applicability (IA) 



Claims NONE 



Claims 1-25 



YES 
NO 



2. citations and explanations (Rule 70.7) 

Claims 1-25 lack novelty under PCT Article 33(2) as being anticipated by LINARES_BARRANCO et al. CMOS Analog Neural 
Network Systems Based on Oscillatory Neurons. IEEE International Symposium on Circuits and Systems. May 1992. As per 
claim 1, LIN ARES- BARRANCO discloses a plurality of n processing elements (figures 7 and 12), a plurality of connectors 
operably coupled with said elements (figures 7 and 12), a conductive medium operably coupled with said connectors (abstract 
and figures 7 and 12) and a forcing apparatus operably coupled with said medium (figures 7 and 12) . As per claim 5, LIN ARES - 
BARRANCO discloses a plurality of n oscillating processing elements (figures 7 and 12), a plurality of no more than n 
connectors (figures 7 and 12), a conductive medium operably coupled with said connectors (abstract and figures 7 and 12) and 
a rhythmic input operably coupled with said medium (figures 7 and 12). As per claim 6, LINARES-BARRANCO discloses a 
plurality of n processing element means (figures 7 and 12), a plurality of connectors operably coupled with said element (figures 
7 and 12), applying an oscillatory signal to each of said element means (figures 7 and 12) and generating said oscillatory signal 
(figures 7 and 12). As per claim 11, LINARES-BARRANCO discloses a number n of oscillating elements (figures 7 and 12), 
a source of a rhythmic elements (figures 7 and 12), a medium interconnecting the source of rhythmic forcing input to each 
oscillating element (figures 7 and 12) and each oscillating element having an oscillating frequency (pages 2236 and 2237 and 
figures 7 and 12). As per claim 19, LINARES-BARRANCO discloses an array of oscillators, at least a plurality of said 
oscillators having differing frequencies (pages 2236 and 2237 and figures 7 and 12), a common conducting medium connected 
to each of the plurality of oscillators (figures 7 and 12) and a source connected to the conducting medium to import oscillator 
signals (pages 2236 and 2237 and figures 7 and 12). As per claim 22, LINARES-BARRANCO discloses operably coupling the 
first element to a medium (pages 2236 and 2237 and figures 7 and 12), operably coupling the second element to said medium 
(pages 2236 and 2237 and figures 7 and 12), operably coupling said medium to a rhythmic input (pages 2236 and 2237 and 
figures 7 and 12) and causing said rhythmic (Continued on Supplemental Sheet.) 
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input to oscillate said medium at a third frequency (pages 2236 and 2237 and figures 7 and 12). As per claim 24, LINARES- 
BARRANCO discloses operably coupling each of the plurality of n elements to a corresponding one of a plurality of no more 
than n connectors (pages 2236 and 2237 and figures 7 and 12), operably coupling each one of said connectors to a conductive 
medium (pages 2236 and 2237 and figures 7 and 12) and operably coupling said medium to rhythmic input (pages 2236 and 
2237 and figures 7 and 12). As per claim 25, LINARES-BARRANCO discloses a number n of active elements and a medium 
having connections to the active elements for application of an input signal thereto, said active elements being phase locked 
loop oscillators (pages 2236 and 2237 and figures 7 and 12). Claims 2-4, 7-10, 12-18. 20, 21 and 23 are also taught by 
LINARES-BARRANCO (see pages 2236 and 2237 and figures 7 and 12). 

Claims 1-24 lack novelty under PCT Article 33(2) as being anticipated by LINSAY et al. The Dynamics of Coupled Nonlinear 
Oscillators. IEEE International Symposium on Circuits and Systems. May 1996. As per claim 1, LINSAY discloses a plurality 
of n processing elements (pages 138 and 139 and figure 1), a plurality of connectors operably coupled with said elements 
(pages 138 and 139 and figure 1), a conductive medium operably coupled with said connectors (pages 138 and 139 and figure 
1) and a forcing apparatus operably coupled with said medium (pages 138 and 139 and figure 1). As per claim 5, LINSAY 
discloses a plurality of n oscillating processing elements (pages 138 and 139 and figure 1), a plurality of no more than n 
connectors (pages 138 and 139 and figure 1), a conductive medium operably coupled with said connectors (pages 138 and 139 
and figure 1) and a rhythmic input operably coupled with said medium (pages 138 and 139 and figure 1). As per claim 6, 
LINSAY discloses a plurality of n processing element means (pages 138 and 139 and figure 1), a plurality of connectors 
operably coupled with said element (pages 138 and 139 and figure 1), applying an oscillatory signal to each of said element 
means (pages 138 and 139 and figure 1) and generating said oscillatory signal (pages 138 and 139 and figure 1). As per claim 
11, LINSAY discloses a number n of oscillating elements (pages 138 and 139 and figure 1), a source of a rhythmic elements 
(pages 138 and 139 and figure 1), a medium interconnecting the source of rhythmic forcing input to each oscillating element 
(pages 138 and 139 and figure 1) and each oscillating element having an oscillating frequency (pages 138 and 139 and figure 
1). As per claim 19, LINSAY discloses an array of oscillators, at least a plurality of said oscillators having differing 
frequencies (pages 138 and 139 and figure 1), a common conducting medium connected to each of the plurality of oscillators 
(pages 138 and 139 and figure 1) and a source connected to the conducting medium to import oscillator signals (pages 138 and 
139 and figure 1). As per claim 22, LINSAY discloses operably coupling the first element to a medium (pages 138 and 139 
and figure 1), operably coupling the second element to said medium (pages 138 and 139 and figure 1), operably coupling said 
medium to a rhythmic input (pages 138 and 139 and figure 1) and causing said rhythmic input to oscillate said medium at a 
third frequency (pages 138 and 139 and figure 1). As per claim 24, LINSAY discloses operably coupling each of the plurality 
of n elements to a corresponding one of a plurality of no more than n connectors (pages 138 and 139 and figure 1), operably 
coupling each one of said connectors to a conductive medium (pages 138 and 139 and figure 1) and operably coupling said 
medium to rhythmic input (pages 138 and 139 and figure 1). Claims 2-4, 7-10, 12-18, 20, 21 and 23 are also taught by 
LINSAY (see pages 138 and 139 and figure 1). 

Claims 19-21 and 25 lack novelty under PCT Article 33(2) as being anticipated by LIU et al. Phrase-Locked Loop with 
Neurocontroller. SICE. July 1998. As per claim 19, LIU discloses an array of oscillators, at least a plurality of said oscillators 
having differing frequencies (pages 1133 and 1134 and figures 1 and 3). a common conducting medium connected to each of 
the plurality of oscillators (pages 1133 and 1134 and figures 1 and 3) and a source connected to the conducting medium to 
import oscillator signals (pages 1133 and 1134 and figures 1 and 3). As per claim 20, LIU discloses feedback circuits 
connected with the medium (pages 1133 and 1134 and figures 1 and 3). As per claim 21, LIU discloses the oscillators are 
phrase locked loops (pages 1133 and 1134 and figures 1 and 3). As per claim 25, LIU discloses a number n of active elements 
and a medium having connections to the active elements for application of an input signal thereto, said active elements being 
phase locked loop oscillators (pages 1133 and 1 134 and figures 1 and 3). 



NEW CITATIONS 

LINSAY et al. The Dynamics of Coupled Nonlinear Oscillators: From Relaxation Oscillatos to Neurons. IEEE Inrenational 
Symposium on Circuits and Systems. 12-15 May 1996, see pages 138 and 139. 

LINNARES-BARRANCO et al. CMOS Analog Neural Network Systems Based on Oscillatory Neurons. IEEE International 
Symposium on Circuits and Systems. 3-6 May 1992. see pages 2236 and 2237 and figures 7 and 12. 



Form PCT/IPEA/409 (Supplemental Box) (July 1998) ★ 



WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 
G06F 15/18, 15/80 



Al 



(11) International Publication Number: 
(43) International Publication Date: 



WO 00/29970 

25 May 2000 (25.05.00) 



(21) International Application Number: PCT/US99/26698 

(22) International Filing Date: 12 November 1999 (12.1 1.99) 



(30) Priority Data: 

60/108,353 



1 3 November 1 998 ( 1 3 . 1 1 .98) US 



(71) Applicant (for all designated States except US): ARIZONA 

BOARD OF REGENTS, a body corporate acting on behalf 
of ARIZONA STATE UNIVERSITY [US/US]; Bank One 
Building, Suite 201, 20 E. University, Tempe, AZ 85282 
(US). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): HOPPENSTEADT, Frank, 
C. [US/US]; 4864 E. Caida Del Sol, Paradise Valley, AZ 
85253 (US). IZHIKEVICH, Eugene [RU/US]; 2700 N. 
Hayden Road #2036, Scottsdale, AZ 85257 (US). 

(74) Agents: MACBLAIN, Thomas, D. et al.; Gallagher & Kennedy, 
2600 N. Central Avenue, Phoenix, AZ 85004 (US). 



(81) Designated States: CN, JP, KR, US, European patent (AT, BE, 
CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, 
NL, PT, SE). 



Published 

With international search report. 



(54) Title: OSCILLATARY NEUROCOMPUTERS WITH DYNAMIC CONNECTIVITY 




(57) Abstract 

A neurocomputer (50) comprises n oscillating processing elements (60A, 60B, 60C, 60D and 60E) that communicate through a 
common medium (70) so that there are required only n connective junctions (80A, 80B, 80C, 80D and 80E). A rhythmic external forcing 
input (90) modulates the oscillatory frequency of the medium (70) which, in turn, is imparted to the n oscillators (60A, 60B, 60C, 60D and 
60E). Any two oscillators oscillating at different frequencies may communicate provided that input's power spectrum includes the frequency 
equal to the difference between the frequencies of the two oscillators in question. Thus, selective communication, or dynamic connectivity, 
between different neurocomputer oscillators occurs due to the frequency modulation of the medium (70) by external forcing. 



\ 



FOR THE PURPOSES OF INFORMATION ONLY 
Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT, 



AL 


Albania 


ES 


Spain 


LS 


Lesotho 


SI 


Slovenia 


AM 


Armenia 


FI 


Finland 


LT 


Lithuania 


SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Luxembourg 


SN 


Senegal 


AU 


Australia 


GA 


Gabon 


LV 


Latvia 


sz 


Swaziland 


AZ 


Azerbaijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


BA 


Bosnia and Herzegovina 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


GH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 


GN 


Guinea 


MK 


The former Yugoslav 


TM 


Turkmenistan 


BF 


Burkina Faso 


GR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BR 


Brazil 


IL 


Israel 


MR 


Mauritania 


UG 


Uganda 


BY 


Belarus 


IS 


Iceland 


MW 


Malawi 


US 


United States of America 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


uz 


Uzbekistan 


CF 


Central African Republic 


JP 


Japan 


NE 


Niger 


VN 


Viet Nam 


CG 


Congo 


KE 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzerland 


KG 


Kyrgyzstan 


NO 


Norway 


zw 


Zimbabwe 


CI 


Cdte d*Ivoire 


KP 


Democratic People's 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Korea 


PL 


Poland 






CN 


China 


KR 


Republic of Korea 


PT 


Portugal 






CU 


Cuba 


KZ 


Kazaksian 


RO 


Romania 






cz 


Czech Republic 


LC 


Saint Lucia 


RU 


Russian Federation 






DE 


Germany 


LI 


Liechtenstein 


SD 


Sudan 






DK 


Denmark 


LK 


Sri Lanka 


SE 


Sweden 






EE 


Estonia 


LR 


Liberia 


SG 


Singapore 







WO 00/29970 



PCT/US99/26698 



OSCILLATORY NEUROCOMPUTERS WITH DYNAMIC 

CONNECTIVITY 

The present application claims priority rights based on U.S. Provisional 
Application Serial No. 60/108,353 filed November 13, 1998. 

FIELD OF THE INVENTION 

The present invention relates generally to computational devices and more 
particularly to a neural network computer requiring a minimal number of connective 
devices between processing elements. 

BACKGROUND OF THE INVENTION 

Artificial neural networks, or neurocomputers, are biologically inspired; that is, 
they are composed of elements that perform in a manner analogous to the most 
elementary functions of the biological neuron. Typically, a neurocomputer is composed 
of a number (n) of processing elements that may be switches or nonlinear amplifiers. 
These elements are then organized in a way that may be related to the anatomy of the 
brain. The configuration of connections, and thus communication routes, between these 
elements represents the manner in which the neurocomputer will function, analogous to 
that of a program performed by digital computers. Despite this superficial resemblance, 
artificial neural networks exhibit a surprising number of the brain's characteristics. For 
example, they learn from experience, generalize from previous examples to new ones, 
and abstract essential characteristics from inputs containing irrelevant data. Unlike a von 
Neumann computer, the neurocomputer does not execute a list of commands (a 
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program). Rather, the neurocomputer performs pattern recognition and associative recall 
via self-organization of connections between elements. 

Artificial neural networks can modify their behavior in response to their 
environment. Shown a set of inputs (perhaps with desired outputs), they self-adjust to 
produce consistent responses. A network is trained so that application of a set of inputs 
produces the desired (or at least consistent) set of outputs. Each such input (or output) 
set is referred to as a vector. Training is accomplished by sequentially applying input 
vectors, while adjusting network weights according to a predetermined procedure. 
During training, the network weights gradually converge to values such that each input 
vector produces the desired output vector. 

Because of their ability to simulate the apparently oscillatory nature of brain 
neurons, oscillatory neurocomputers are among the more promising types of 
neurocomputers. Simply stated, the elements of an oscillatory neurocomputer consist of 
oscillators rather than amplifiers or switches. Oscillators are mechanical, chemical or 
electronic devices that are described by an oscillatory signal (periodic, quasi-periodic, 
almost periodic function, etc.). Usually the output is a scalar function of the form 
V(cDt+<j)) where V is a fixed wave form (sinusoid, saw-tooth or square wave), co is the 
frequency of oscillation, and § is the phase deviation (lag or lead). 

Recurrent neural networks have feedback paths from their outputs back to their 
inputs. As such, the response of such networks is dynamic in that after applying a new 
input, the output is calculated and fed back to modify the input. The output is then 
recalculated, and the process is repeated again and again. Ideally, successive iterations 
produce smaller and smaller output changes until eventually the outputs become 
constant. To properly exhibit associative and recognition properties, neural networks, 



# 
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such as is required by Hopfield's network, must have a fully connected synaptic matrix. 
That is, to function optimally, recurrent network processing elements must communicate 
data to each other. Although some prototypes have been built, the commercial 
manufacture of such neurocomputers faces a major problem: A conventional recurrent 
5 neurocomputer consisting of n processing elements requires n 2 connective junctions to be 
fully effective. The terms connector or connective junction, as used herein throughout, 
are defined as a connective element that enables one processing element to receive as 
input data output data produced by itself or any other one processing element. For large 
n this is difficult and expensive. 

10 Accordingly, a need exists for a neurocomputer with fully recurrent capabilities 

and requiring a minimal number of connective devices between processing elements. 



15 exhibits pattern recognition and associative recall capabilities while requiring only n 
connective junctions for every n processing elements employed thereby. 

In a preferred embodiment of the invention, the neurocomputer comprises n 
oscillating processing elements that can communicate through a common medium so that 
there are required only n connective junctions. A rhythmic external forcing input 

20 modulates the oscillatory frequency of the medium which, in turn, is imparted to the n 
oscillators. Any two oscillators oscillating at different frequencies may communicate 
provided that the input's power spectrum includes the frequency equal to the difference 
between the frequencies of the two oscillators in question. Thus, selective 
communication, or dynamic connectivity, between different neurocomputer oscillators 

25 occurs due to frequency modulation of the medium by external forcing. 



SUMMARY OF THE INVENTION 



In accordance with the present invention, a neurocomputer is disclosed that 
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BRIEF DESCRIPTION OF THE 
DRAWING 

FIG. 1 is a schematic diagram of a prior art recurrent neural network employing 
five neural processing elements. 

FIG. 2 is a schematic diagram of a neural network according to principles of the 
present invention and employing five neural processing elements. 

FIG. 3 is a diagrammatic illustration of results obtained through simulation of the 
neurocomputer according to principles of the present invention using a phase deviation 
model and Hebbian learning rule with parameters: n = 60, / e [0, 10]. 

FIG. 4 is a schematic block diagram of a phase-locked loop. 

FIG. 5 is a diagrammatic illustration of the relationship between demodulated 
output voltage and input frequency and phase of a phase-locked loop as depicted in FIG. 
4. 

FIG. 6 is a schematic block diagram of a neural network according to principles 
of the present invention employing two phase-locked loops as depicted in FIG. 4. 

FIG. 7 is a diagrammatic illustration of one frequency multiplication performed 
in the neural network depicted in FIG. 6. 

FIG. 8 is a schematic diagram showing the circuit components of the neural 
network of FIG. 6 according to principles of the present invention. 

FIG. 9 is a schematic block diagram of a five-oscillator neural network with 
associated function generator and oscilloscopes connected for testing. 
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FIGS. 10A-10E are oscilloscope traces of the oscillator responses of the network 
of FIG. 9 with a signal sin t impressed on the input. 

FIGS. 1 1 A-l IE are oscilloscope traces of the oscillator responses of the network 
of FIG. 9 with a signal sin 2t impressed on the input. 
5 FIGS. 12A-12E are oscilloscope traces of the oscillator responses of the network 

of FIG. 9 with a signal sin 3t impressed on the input. 

FIGS. 13A-13E are oscilloscope traces of the oscillator responses of the network 
of FIG. 9 with a signal sin lOt impressed on the input. 

1 0 DESCRIPTION OF THE PREFERRED EMBODIMENT 

FIG. 1 schematically illustrates a conventional recurrent neurocomputer 10 

comprising n (in this case, ?7=5) neural processing elements 20. Elements 20 may 
comprise switches, amplifiers, oscillators or any other suitable neurocomputer element 
type known in the art. In order for each of elements 20 to communicate with the others 
15 of elements 20, neurocomputer 10 necessarily includes n (in this case, n 2 =25) 
connective junctions 30 to which conductors 40 are attached. As can be observed, where 
the number n of elements 20 grows large, the implementation of such a neurocomputer 
becomes prohibitively difficult, from both cost and practicability standpoints. 

FIG. 2 schematically illustrates a neurocomputer 50 according to principles of the 
20 present invention. Neurocomputer 50 comprises a finite number n (in this case, n=S) 
oscillatory neural processing elements 60A, 60B, 60C, 60D and 60E. Elements 60A, 
60B, 60C, 60D and 60E can comprise voltage-controlled oscillators, optical oscillators, 
lasers, microelectromechanical systems, Josephson junctions, macromolecules, or any 
other suitable oscillator known in the art. Each element 60A, 60B, 60C, 60D and 60E 
25 oscillates at a particular frequency that may or may not be the same frequency as that of 
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the others of elements 60A, 60B, 60C, 60D and 60E. In its most general sense, the 
neurocomputer 50 further comprises a medium 70 connected to each of elements 60A, 
60B, 60C, 60D and 60E by means of connective junctions 80A, 80B, 80C, 80D and 80E, 
respectively. Medium 70 may comprise a unitary body or multiple connected bodies. 
5 Neurocomputer 50 further comprises a rhythmic forcing signal source 90 able to apply a 
modulated oscillatory frequency to medium 70 by means of a connection 100. 
Specifically, the medium 70 can be a conductive medium electrically connected to the 
oscillators 60A, 60B, 60C, 60D and 60E by conductive connection junctions 80A, 80B, 
80C, 80D and 80E. The rhythmic forcing signal source 90 can be an electrical signal 
10 generator such as a frequency modulated transmitter connected by a conductive 
connection 100 to the medium 70. 

In operation, any two elements, such as 60B and 60E, can be said to 
communicate to each other if changing the phase deviation of one influences the phase 
deviation of the other. Such is the case if the two elements oscillate at the same 
15 frequency. Accordingly, if elements 60B and 60E oscillate at the same frequency, they 
will communicate in such manner. 

If elements 60B and 60E oscillate at different frequencies, they will not 
communicate in such manner. However, by causing input signal source 90 to apply a 
uniform oscillatory signal multiplicatively to elements 60A, 60B, 60C, 60D and 60E by 
20 way of medium 70, any two oscillators, such as 60B and 60E, can be made to 
communicate by filling the frequency gap between them. That is, the uniform oscillatory 
signal must include a frequency equal to the difference between the respective 
frequencies of elements 60B and 60E. Accordingly, if elements 60B and 60E are 
oscillating at two different frequencies, say col and co2, then applying the time (r) 
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dependent voltage signal a(0=cos(col-co2)f to medium 70 enables elements 60B and 60E 
to communicate data to each other. 



Mathematical analysis of the said neurocomputer architecture, which is based on 
the theory developed by F. C. Hoppensteadt and E. M. Izhikevich (Oscillatory 
5 neurocomputers with dynamic connectivity, Physical Review Letters 82(1999)2983- 
2986) shows that the neurocomputer dynamic is equivalent to a fully connected Hopfield 
network (J.J. Hopfield, Neural networks and physical systems with emergent collective 
computational abilities, Proceedings of National Academy of Sciences (USA) 
79(1982)2554-2558). In particular, we use the well-known Hebbian learning rule 

10 (D.O.Hebb, The Organization of Behavior, J. Wiley, New York, 1949; and S. Grossberg, 
Non-linear neural networks: Principles, mechanisms and architectures, Neural Networks 
1(1988)17-61) to show that a network of n ~ 60 oscillators can memorize and 
successfully retrieve through associative recall three patterns corresponding to the 
images "0", "1", "2", as we illustrate in Figure 3. Thus, the neurocomputer can act as a 

15 classical fully connected Hopfield network despite the fact that it has only n 
interconnections. 

As discussed below, a neurocomputer according to principles of the present 
invention may be comprised mainly of phase-locked loops, amplifiers, and band-pass 
filters. A schematic of such a neurocomputer is shown in FIG. 8. In this discussion, 
20 emphasis will be placed on the operation of phase-locked loops, which are ideally 
designed to perform frequency demodulation and frequency multiplication. Because of 
these qualities, they are highly suited for simulating neuron interaction. 
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A schematic of the major components of a phase locked loop ("PLL") 110 is 
shown in FIG. 4. The major components include a phase detector 120, low-pass filter 
130, unity amplifier 140, and a voltage controlled oscillator ("VCO") 150. Phase locked 
loops use a feedback loop to produce a replica of an input signal's frequency. They are 
similar to operational amplifiers ("op-amps") in that an op-amp amplifies the voltage 
difference between input signals. The difference is that a PLL amplifies the frequency 
difference of the inputs and sets them equal to each other, so that the internally generated 
signal in the VCO 150 is an exact replica of the input signal (pin 4 of the PLL). Once 
this has occurred the PLL 110 is said to be in the "locked on" state. When the two 
signals are "locked on," any change in the input's frequency is detected by the phase 
detector 120 as an error signal. This error signal is applied to the internal signal, which 
is a replica of the input, so that it will match the input signal's frequency. The error 
signal is essentially the phase difference in the signal, which is the information 
waveform. The encoded information is extracted from pin 7 of the PLL 110. By 
implementing the above technique, frequency demodulation is performed using PLLs. 

PLLs may be set up to perform frequency multiplication. This is accomplished 
by placing an open circuit between pins 3 and 4 in FIG. 4 and inputting a second source 
at pin 3. Since the phase detector 120 of PLL 1 10 is classified as type 1, it has a highly 
linear XOR gate and a built-in four-quadrant multiplier. The four-quadrant multiplier 
allows PLL 1 10 to perform frequency multiplication very accurately. A PLL connected 
in this manner produces an output that is the frequency multiplication of the two inputs. 

Before simulating neuron activity using phase locked loop circuitry, one first 
establishes the free running frequency, the capture-range, and the lock-range of the PLL. 
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The free running frequency (f 0 ) is ideally the center frequency level of the signal that is 
to be demodulated. The value for the free running frequency is obtained from 

fo= 1.2/(4#R1*C1) (/) 

It should be mentioned that the resistance Rl and the capacitance CI correspond 
to the values of resistor Rl and capacitor CI in FIG. 4. The capture-range (f c ) is the 
frequency range over which the PLL will try to lock on to an input's frequency. The 
following formula may be used to determine the capture-range. 

fc = l/(2#7r>V((2*7r*fL)/(3.6#1000*C2)) (ii) 

where C2 is the capacitance of the similarly designated capacitor in FIG. 5 and f L is the 
lock-range. 

By evaluating the formula for the capture range, one can see that the capture 
range is limited by low pass filter time constant. The lock-range (fO is the range over 
which the PLL will remain in the locked on state. This range is generally larger than the 
capture-range and can be increased by increasing Vcc of the PLL as shown in the 
following equation. 

ft = 8*fo/Vcc (Hi) 

After establishing the free running, lock and capture frequencies, it should be 
determined if there exists a linear relationship between the input frequency and phase, 
and the demodulated output voltage. This linear relationship can be determined and 
demonstrated as shown in FIG. 5. 

Communication can occur when a signal is outside the capture range if it is 
conditioned by another signal. This can be demonstrated by implementing the multiple 
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PLL circuit 170 as shown in FIG. 6. The key to designing the circuit depicted in FIG. 6 
is the ability to obtain the sum and difference of two input frequencies, which can be 
accomplished through multiplication. 

In order to implement the multiplication operation as shown by the multiplication 
circle 171 or 172, one should understand the following theory: 

cos(coc) cos(com) = (l/2)»[cos(coc - com) + cos(coc + com)] 

=> Fourier Transform => 
(l/4>[8(f + (fc - fin)) + S(f + (fc + fin)) + S(f- (fc - fin)) + 6(f- (fc + fin))] 

FIG. 7 shows what occurs when multiplying 8kHz and 42kHz, as at the multiplier 
171 in FIG. 7. As seen in FIG. 8, these 8kHz and 42kHz components are present in the 
output 172 of the multiplier 171. Also present are 50kHz and 34 kHz, the sum and 
difference of 8 kHz and 42 kHz, respectively. Interfering harmonics are also present. 
Here, 8 kHz and 42 kHz were chosen to obtain an adequate separation between the 
harmonics and the desired frequency components. 

To resolve the problem of harmonic frequencies output by the multipliers 171, 
173, band pass filters were placed in the circuit. The filter was comprised of an inductor 
and capacitor. To isolate the single frequency desired, the following formula was used: 

f = 1/(V(L*02*ti)) <7v) 

This formula was used to choose the desired inductor and capacitor needed. A problem 
resulting from the addition of the band pass filter was a voltage drop of the input signal. 
To compensate for this drop, amplifiers were inserted into the circuit to raise the voltage 
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back up to the desired magnitude, 
indicated at 175, 176 of FIG. 6. 
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The band pass filter amplifier circuits thus added are 



By modulating the 8 kHz carrier frequency of the function generator 178 with a 
100 Hz sine wave modulation and multiplying the modulated signal with a 42 kHz 
carrier frequency, the PLL 181 was able to demodulate the input signal and output the 
100 Hz information signal. Similarly, the PLL 182 was able to demodulate the 100 Hz 
information signal. Testing of the circuit depicted in FIG. 6 demonstrates that 
communication can still occur even if a signal is outside the capture range of a PLL if the 
information signal is combined with another carrier signal. 

In FIG. 8, the circuit shown in block diagram form in FIG. 6 is schematically 
shown in greater detail. The multipliers 172 and 173 are LM 565 phase locked loops 
from National Semiconductor. National Semiconductor op amps LM 324 are used in the 
band pass filter and amplification stages 175 and 176 along with the inductor and 
capacitor filtering circuit elements of the values shown. The phase locked loops 181 and 
182 connected as oscillators employ the LM 565 phase locked loops from National 
Semiconductor. VCd, from the PLL oscillator 181 at the upper right, is fed back to the 
multiplier 173 at the lower left and VCC 2 is fed back from the PLL oscillator 182 at the 
lower right to the multiplier 1 72 at the upper left. 

In FIG. 9, a five-element neural network 190 is shown. Five phase locked loop 
oscillators 191, 192, 193, 194 and 195 are tuned at frequencies W,, W 2 , W 3 , W 4 , W 5 , 
respectively, where the ratio of frequencies is W, : W 2 :W 3 :W 4 :W5 = 1 :2:3:4:5. These 
oscillators are forced by a common function generator 198. The function generator is 
connected to the oscillators via the conductors 200 and 202 serving as the conductive 
medium and connectors, respectively, of the neural network 190 previously discussed. 



11 



WO 00/29970 PCT/US99/26698 

Oscilloscopes 21 1, 212, 213, 214, and 215 are connected to the oscillators as illustrated 
to demonstrate the output signals of the oscillators responsive to various inputs from the 
function generator 198. A summing circuit 21 7 is connected into a feedback loop 218 
common to each of the oscillators. 

FIGS. 10-13 illustrate the responses of the oscillators 191-195, respectively, to 
four input signals generated by the function generator 198. FIG. 10 shows the traces of 
the oscilloscopes 211-215 corresponding to the oscillating signals of oscillators 191-195, 
respectively, when the forcing voltage from the function generator 198 is sin t. None of 
the oscillators 191-195 are in communication and the oscillator signals are unrelated, as 
shown by the traces 10A-10E. 

With a function generator input sin 2t, the oscillators respond with signals as 
shown by the traces of FIGS. 1 1 A to 1 IE. Here, the oscillators 191, 193 and 195 
communicate, producing the oscilloscope traces of FIGS. 1 1 A, 1 1C and 1 IE, and the 
oscillators 192 and 194 communicate producing the traces of FIGS. 1 IB and 1 ID. 

With an input signal sin 3t impressed by the function generator 198, the 
oscillators 191-195 produce in the oscilloscopes 21 1-215 the traces shown at FIGS. 12A- 
12E, respectively. Oscillators 191 and 194 communicate, producing the traces of FIGS, 
12A and 12D. Oscillators 192 and 195 communicate producing the traces shown at 
FIGS. 12B and 12E. The oscillator 193 is not in communication with any other of the 
oscillators and it produces the trace shown at FIG. 12C. 

In FIGS. 13A-13E, shown are traces illustrating the response of the oscillators 
191-195 when a signal sin lOt is impressed by the function generator 198. None of the 
oscillators here are communicating. 
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A further, more generalized example of implementation of the present invention 
will now be described using a network of n voltage controlled oscillators (known as 
Kuramoto's phase model) and represented by: 



where ^ - s ' is the phase of the /th oscillator, a(f) is the external input and e « 1 is 
the strength of connections. We require that all differences ^» " be different when / * 
J- 

10 

(i) Averaging 



(1) 



5 



Let 



then 



n 



15 



<pi = ca(t) ]T siniiClj - Q^t + Vj - Vi ) . ( 2 ) 



One can average this system to obtain 



<fii-cEi(<fi X ,...,^ n ) + 0 {e) , 
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where 

is the average of the right-hand side of (2). 
(ii) Quasiperiodic External Input 

Now suppose we are given a matrix of connections C = (Cjj). Let 
a(i) = ao + T *j «*({<>, - n t } 0 (4; 

1=1 J=l 

be a time dependent external input, which is a quasiperiodic function of t. Since all 
differences are different for all i and j\ it is easy to verify that 

If we denote Sy = (Cy + c 7 /)/2, use the slow time r = et, and disregard the small-order 
term o(e), then we can rewrite system (2) in the form 



>=1 



sin(V ; - Vi) : 



(5) 
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where =d/dr. We see that the external input of the form (4) can dynamically connect 
any two oscillators provided that the corresponding dj is not zero. 

(iii) Chaotic External Input 

In general, the external input a(t) can be chaotic or noisy. It can dynamically connect the 
/th and the yth oscillators if its Fourier transform has a non-zero entry corresponding to 
the frequency = - fi > since the average, H h would depend on the phase difference 
^ in this case. 

(iv) Oscillatory Associative Memory 

Since the connection matrix S — (Sjj) is symmetric, the phase model (5) is a gradient 
system. Indeed, it can be written in the form 

where 

j n n 
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is a gradient function. The vector of phase deviations ^ = (v*"*- »^r*) € T" a i wa y S 

converges to an equilibrium on the n-torus e as shown in FIG. 3. System (5) has 
multiple attractors and Hopfield-Grossberg-like associative properties as also shown in 
FIG. 3. Therefore, system (1) with external forcing has oscillatory associative memory. 

5 

(v) Hebbian Learning Rule 



Suppose we are given a set of m key vectors to be memorized 



where St ~~ ^ means that the z'th and theyth oscillators are in-phase = and 

~ means they are anti-phase — v> ~^ ^ Hebbian learning rule of the form 



Jfc=i 

15 

is the simplest one among many possible learning algorithms. To get (5) it suffices to 
apply the external input of the form (4) with c,y = Sy for all i and j. 

(vi) Initializing the Network 

20 

To use the proposed neurocomputer architecture to implement the standard Hopfield- 
Grossberg paradigm, as we illustrate in FIG. 3, we need a way to present an input image 

as an initial condition ^ K $\ and to read the output from the network. While the latter 

16 
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task poses no difficulty and can be accomplished using Fourier analysis of the "mean 
field activity," the former task requires some ingenuity since we do not have direct 
access to the oscillators. 

Suppose we are given a vector f° - R" to be recognized. Let us apply the external input 
a(t) c, ; = with for a certain period of time. This results in the phase 

deviation system of the form 

10 

n 

It is easy to check that if, = i then <?i(0-Vj(0 0. and if $Hj = -1 then 

1 5 sjift) - «?j(t)- -* ~ ^ or a " z anc * ^* Thus, the network activity converges to the equilibrium 

having phase relations defined by the vector ^ , as shown in FIG. 3. When we restore 
the original external input a(t), which induces the desired dynamic connectivity, the 

recognition starts from the input image N . (We added noise to the image s shown in 
FIG. 3 to enhance the effect of convergence to an attractor during recognition.) 

20 Although the invention has been described in terms of the illustrative 

embodiment, it will be appreciated by those skilled in the art that various changes and 
modifications may be made to the illustrative embodiment without departing from the 
spirit or scope of the invention. It is intended that the scope of the invention not be 
limited in any way to the illustrative embodiment shown and described but that the 

25 invention be limited only by the claims appended hereto. 
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WHAT IS CLAIMED IS: 

1 . A neurocomputer comprising: 

a plurality of n processing elements; 

a plurality of connectors operably coupled with said elements; 
a conductive medium operably coupled with said connectors; and 
a forcing apparatus operably coupled with said medium. 

2. The neurocomputer of claim 1, wherein: 

said plurality of connectors comprises no more than n connectors, 

each of 

said connectors being operably coupled with a corresponding one of said elements. 

3. The neurocomputer of claim 1 , wherein: 

said forcing apparatus comprises a rhythmic input. 

4. The neurocomputer of claim 1 , wherein: 

said elements comprise oscillators. 

5. A neurocomputer comprising: 

a plurality of n oscillating processing elements; 
a plurality of no more than n connectors, each of said connectors 
being operably coupled with a corresponding one of said elements; 

a conductive medium operably coupled with said connectors; and 
a rhythmic input operably coupled with said medium. 

6. A neurocomputer comprising: 

a plurality of n processing element means; 

a plurality of connectors operably coupled with said element 

means; 
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means for simultaneously applying an oscillatory signal to each of 
said element means via said connectors; and 

means for generating said oscillatory signal operably coupled with 
said means for applying. 
5 7. The neurocomputer of claim 6, wherein: 

said plurality of connectors comprises n connectors, each of said 
connectors being operably coupled with a corresponding one of said element means. 
8. The neurocomputer of claim 6, wherein: 

said element means comprise oscillators. 
10 9. The neurocomputer of claim 6, wherein: 

said means for applying comprises a conductive medium. 

10. The neurocomputer of claim 6, wherein: 

said means for generating comprises a rhythmic input. 

11. An oscillatory neurocomputer comprising: 
15 a number n of oscillating elements 

a source of a rhythmic forcing input, 

a medium interconnecting the source of rhythmic forcing input to 
each 

oscillating element, 

20 each oscillating element having an oscillating frequency, 

the oscillating frequency f] of at least one of the oscillating 
elements differing from the oscillating frequency f2 of at least one other of the oscillating 
elements, 



19 





WO 00/29970 



PCT/US99/26698 



the source of a rhythmic forcing input producing an input 



10 



15 



of a third frequency f 3 , establishing communication between the at least one oscillating 
element and the at least one other oscillating element. 

12. An oscillatory neurocomputer according to claim 1 1 , wherein f 3 is 
substantially the difference between fi and f 2 . 

13. An oscillatory neurocomputer according to claim 11, further comprising a 
number ni of connections of the source of a rhythmic forcing input to the oscillating 
elements, wherein 



14. An oscillatory neurocomputer according to claim 12, further comprising a 
number ni of connections of the source of a rhythmic forcing input to the oscillating 
elements, wherein 



15. An oscillatory neurocomputer according to claim 11, wherein the 
oscillating elements are electronic oscillators, the source of a rhythmic forcing input is a 
function generator and the interconnecting medium is an electrically conductive medium 
electrically connecting the source of a rhythmic forcing input to the oscillators. 

16. An oscillatory neurocomputer according to claim 15, wherein the function 
generator provides a forcing signal having a carrier frequency and information content 
modulating the carrier frequency, the oscillators responding to the impression of the 
forcing signal onto the conductive medium to produce information content modulation 
substantially the same as that of the conductive medium. 

17. An oscillatory neurocomputer according to claim 11, wherein the number 



ni < n. 



n of 
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oscillating elements is greater than two, a first subset of the oscillating elements 
communicate at a frequency f3 of rhythmic forcing input from the source, and at least one 
second subset of the oscillating elements communicate at least one further frequency £4 
of rhythmic forcing input from the source. 

1 8. An oscillatory neurocomputer according to claim 1 5, wherein content 
varying one 

oscillator from its oscillating frequency is communicated to and varies from its 
oscillating frequency another oscillator in communication with the one oscillator. 

1 9. A neurocomputer including: 

(a) an array of oscillators, at least a plurality of said oscillators having 
differing frequencies, 

(b) a common conducting medium connected to each of the plurality 
of 

oscillators, 

(c) a source connected to the conducting medium to impart oscillator 
signals 

of various frequencies to the conducting medium, the signals of various frequencies 
including frequencies effective to bring two or more of the oscillators into 
communication. 

20. An oscillatory neurocomputer according to claim 19, wherein the 
oscillators include feedback circuits connected with the medium. 

21. An oscillatory neurocomputer according to claim 20, wherein the 
oscillators are phase locked loops. 
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22. A method of enabling communication of a characteristic between a first 
processing element oscillating at a first frequency and a second processing element 
oscillating at a second frequency different from the first frequency, the method 
comprising the steps of: 
5 operably coupling the first element to a medium; 

operably coupling the second element to said medium; 
operably coupling said medium to a rhythmic input; and 
causing said rhythmic input to oscillate said medium at a third 

frequency. 

10 23. The method of claim 22, wherein: 

said third frequency comprises a frequency substantially equal to 

the 

difference between the first frequency and the second frequency. 

24. A method of enabling communication of a characteristic between a 
15 plurality of n oscillating processing elements comprising the steps of: 

operably coupling each of the plurality of n elements to a 
corresponding one of a plurality of no more than n connectors; 

operably coupling each one of said connectors to a conductive 

medium; and 

20 operably coupling said medium to a rhythmic input. 

25. In a neurocomputer, a number n of active elements and a medium having 
connections to the active elements for application of an input signal thereto, said active 
elements being phase locked loop oscillators. 
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